Image Analytics¶
The marketing team at Coca-Cola would like to perform visual listening by investigating the images taken and shared by consumers on Flickr.com. The user-generated images reflect consumers’ impression of the brand (or marketing campaign) more truly than self-reported statements. The members of the team are aware that they can create better future ad copies by learning which images would and would not become popular. Furthermore, by studying the link between image attributes and their popularity, the team can have a better idea of (a) which ad features are appreciated by consumers, and (b) how to design more appealing advertisements in the future.
Learning objectives
At the end of this computer application, you will be able to:
Use convolutional neural networks (CNNs) to classify consumer-uploaded images of the Coca-Cola brand on Flickr.
Illustrate which ad features of Coca-Cola lead to higher popularity among consumers, using a small sample of images.
Preparation and set-up¶
We need to make sure that all our files are organized and our Python environment is set up. To do so, we execute the following steps:
Create a folder (e.g. chapter 10) on our device.
Download the data and all the other files needed for this session into the folder we have created. The files are available on the book's website.
Next, we launch a JupyterLab session from the Anaconda environment, set the working directory, and load the dataset into Jupyter Notebook for analysis. ed.
Setting up the working directory¶
The following code chunk sets up the working directory. Make sure to change the file path below to your own.
#Find the current working directory
import os
cwd = os.getcwd()
print(cwd)
#Change the working directory
os.chdir('C:/Users/gyildiri/Downloads/chapter 10')
print(os.getcwd())
C:\Users\gyildiri\Downloads\chapter 10 C:\Users\gyildiri\Downloads\chapter 10
Installing the Python libraries¶
%pip install pandas tabulate
import pandas as pd
from tabulate import tabulate
Requirement already satisfied: pandas in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (2.1.4)Note: you may need to restart the kernel to use updated packages. Requirement already satisfied: tabulate in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (0.9.0) Requirement already satisfied: numpy<2,>=1.23.2 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from pandas) (1.26.4) Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from pandas) (2.8.2) Requirement already satisfied: pytz>=2020.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from pandas) (2023.3.post1) Requirement already satisfied: tzdata>=2022.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from pandas) (2023.3) Requirement already satisfied: six>=1.5 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
Before we roll out the analyses, we need to ensure reproducibility in our models that involve random number generation. For that, we use the following code snippet.
import numpy as np
np.random.seed(2311)
Data¶
We obtain the image data on Coca-Cola ads from Flickr, a famous online portal where people share photos that they take themselves. We searched for all images about Coca-Cola, shared by users during the first quarter of 2021. Although we use a small sample of data (20 pictures), the Python code we present here can easily be extended to large datasets.The basic information about the images is summarized in the data file ‘flickr_data.xlsx’. In the following code chunk, we load the excel file to explore the features of this data set.
# Load the data
data = pd.read_excel("flickr_data.xlsx")
# Display the data using tabulate
print(tabulate(data, headers='keys', tablefmt='fancy_grid'))
╒════╤═════════════════════╤════════════════╤═════════════════════╤═══════════════════╤══════════════════════╤═══════════╤════════════════╤═════════════════╤════════════════════════╤═══════════════════════════╕ │ │ Original Image ID │ No. of Views │ No. of Favourites │ No. of Comments │ Index for Analysis │ Popular │ Colour Image │ Human Centric │ Direct Logo Exposure │ Direct Product Exposure │ ╞════╪═════════════════════╪════════════════╪═════════════════════╪═══════════════════╪══════════════════════╪═══════════╪════════════════╪═════════════════╪════════════════════════╪═══════════════════════════╡ │ 0 │ 20 │ 9535 │ 52 │ 5 │ Popular 1 │ 1 │ 1 │ 1 │ 1 │ 0 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 1 │ 18 │ 2702 │ 165 │ 12 │ Popular 2 │ 1 │ 1 │ 0 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 2 │ 3 │ 2427 │ 144 │ 3 │ Popular 3 │ 1 │ 0 │ 0 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 3 │ 5 │ 2374 │ 62 │ 6 │ Popular 4 │ 1 │ 1 │ 1 │ 1 │ 0 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 4 │ 15 │ 1770 │ 9 │ 1 │ Popular 5 │ 1 │ 1 │ 1 │ 0 │ 0 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 5 │ 4 │ 1539 │ 35 │ 2 │ Popular 6 │ 1 │ 1 │ 0 │ 1 │ 0 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 6 │ 21 │ 1512 │ 46 │ 16 │ Popular 7 │ 1 │ 1 │ 0 │ 1 │ 0 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 7 │ 13 │ 1272 │ 20 │ 2 │ Popular 8 │ 1 │ 1 │ 0 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 8 │ 19 │ 1043 │ 140 │ 7 │ Popular 9 │ 1 │ 1 │ 0 │ 1 │ 0 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 9 │ 14 │ 1009 │ 35 │ 9 │ Popular 10 │ 1 │ 1 │ 0 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 10 │ 7 │ 699 │ 11 │ 0 │ Unpopular 1 │ 0 │ 1 │ 0 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 11 │ 17 │ 631 │ 5 │ 0 │ Unpopular 2 │ 0 │ 1 │ 0 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 12 │ 12 │ 11 │ 0 │ 0 │ Unpopular 3 │ 0 │ 1 │ 0 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 13 │ 11 │ 613 │ 8 │ 0 │ Unpopular 4 │ 0 │ 1 │ 1 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 14 │ 1 │ 570 │ 27 │ 5 │ Unpopular 5 │ 0 │ 1 │ 0 │ 1 │ 0 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 15 │ 6 │ 538 │ 40 │ 11 │ Unpopular 6 │ 0 │ 1 │ 1 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 16 │ 8 │ 414 │ 33 │ 0 │ Unpopular 7 │ 0 │ 1 │ 0 │ 1 │ 0 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 17 │ 16 │ 94 │ 1 │ 0 │ Unpopular 8 │ 0 │ 1 │ 0 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 18 │ 10 │ 83 │ 0 │ 0 │ Unpopular 9 │ 0 │ 1 │ 0 │ 1 │ 1 │ ├────┼─────────────────────┼────────────────┼─────────────────────┼───────────────────┼──────────────────────┼───────────┼────────────────┼─────────────────┼────────────────────────┼───────────────────────────┤ │ 19 │ 9 │ 43 │ 0 │ 0 │ Unpopular 10 │ 0 │ 1 │ 0 │ 1 │ 1 │ ╘════╧═════════════════════╧════════════════╧═════════════════════╧═══════════════════╧══════════════════════╧═══════════╧════════════════╧═════════════════╧════════════════════════╧═══════════════════════════╛
In the output, the column ‘popular’ shows whether the image is considered ‘popular’ or ‘unpopular’. To classify (or pre-label) images into ‘popular’ and ‘unpopular’ class, we use the number of likes as a proxy for popularity, randomly select 20 images, and rank them by the number of likes. We label the top 10 images as ‘popular’, and the rest as ‘unpopular’.
We summarize the descriptions of the other variables by running the code chunk below.
import pandas as pd
from tabulate import tabulate
# Creating a DataFrame for variable explanations
var_exp_data = {
"Item": ["Original ID", "No. of Views", "No. of Favourites", "No. of Comments", "Index for Analysis", "Popular", "Colour Image", "Human Centric", "Direct Logo Exposure", "Product Exposure"],
"Description": ["Original ID of each image in the database", "Total number of views of each image", "Total number of favourites received of each image", "Total number of comments of each image", "Image index (i.e., after labelling) used for analysis", "Image label (Popular = 1, Unpopular = 0)", "Dummy variable indicating if the image is colour (1) or black-and-white (0)", "Dummy variable indicating if the image is human-centric (1) or not (0)", "Dummy variable indicating if the Coca-Cola logo was displayed in the image (1 = Yes, 0 = No)", "Dummy variable indicating if the product (i.e., Coke) was displayed in the image (1 = Yes, 0 = No)"]
}
var_exp_df = pd.DataFrame(var_exp_data)
# Display the variable explanations using tabulate
print(tabulate(var_exp_df, headers='keys', tablefmt='fancy_grid'))
╒════╤══════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════╕ │ │ Item │ Description │ ╞════╪══════════════════════╪════════════════════════════════════════════════════════════════════════════════════════════════════╡ │ 0 │ Original ID │ Original ID of each image in the database │ ├────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ 1 │ No. of Views │ Total number of views of each image │ ├────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ 2 │ No. of Favourites │ Total number of favourites received of each image │ ├────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ 3 │ No. of Comments │ Total number of comments of each image │ ├────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ 4 │ Index for Analysis │ Image index (i.e., after labelling) used for analysis │ ├────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ 5 │ Popular │ Image label (Popular = 1, Unpopular = 0) │ ├────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ 6 │ Colour Image │ Dummy variable indicating if the image is colour (1) or black-and-white (0) │ ├────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ 7 │ Human Centric │ Dummy variable indicating if the image is human-centric (1) or not (0) │ ├────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ 8 │ Direct Logo Exposure │ Dummy variable indicating if the Coca-Cola logo was displayed in the image (1 = Yes, 0 = No) │ ├────┼──────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ 9 │ Product Exposure │ Dummy variable indicating if the product (i.e., Coke) was displayed in the image (1 = Yes, 0 = No) │ ╘════╧══════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════╛
Next, we install the Python library Pillow and unzip the images that are located in our working directory.
%pip install Pillow
Requirement already satisfied: Pillow in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (10.2.0) Note: you may need to restart the kernel to use updated packages.
!unzip images.zip
'unzip' is not recognized as an internal or external command, operable program or batch file.
Then, we create a list of images using the code below.
from PIL import Image
# Save image names in a list
pics = [
"images/popular_1.jpeg", "images/popular_2.jpeg", "images/popular_3.jpeg", "images/popular_4.jpeg", "images/popular_5.jpeg",
"images/popular_6.jpeg", "images/popular_7.jpeg", "images/popular_8.jpeg", "images/popular_9.jpeg", "images/popular_10.jpeg",
"images/unpopular_1.jpeg", "images/unpopular_2.jpeg", "images/unpopular_3.jpeg", "images/unpopular_4.jpeg", "images/unpopular_5.jpeg",
"images/unpopular_6.jpeg", "images/unpopular_7.jpeg", "images/unpopular_8.jpeg", "images/unpopular_9.jpeg", "images/unpopular_10.jpeg"
]
mypic = []
# Load files into list using a for loop
for pic in pics:
mypic.append(Image.open(pic))
We install the Python library matplotlib to import the images.
%pip install matplotlib
Requirement already satisfied: matplotlib in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (3.8.0) Requirement already satisfied: contourpy>=1.0.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from matplotlib) (1.2.0) Requirement already satisfied: cycler>=0.10 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from matplotlib) (0.11.0) Requirement already satisfied: fonttools>=4.22.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from matplotlib) (4.25.0) Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from matplotlib) (1.4.4) Requirement already satisfied: numpy<2,>=1.21 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from matplotlib) (1.26.4) Requirement already satisfied: packaging>=20.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from matplotlib) (23.1) Requirement already satisfied: pillow>=6.2.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from matplotlib) (10.2.0) Requirement already satisfied: pyparsing>=2.3.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from matplotlib) (3.0.9) Requirement already satisfied: python-dateutil>=2.7 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from matplotlib) (2.8.2) Requirement already satisfied: six>=1.5 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0) Note: you may need to restart the kernel to use updated packages.
We display the images using the code chunk below.
from PIL import Image
import matplotlib.pyplot as plt
# Load two sample images
popular_sample = Image.open("images/popular_4.jpeg")
unpopular_sample = Image.open("images/unpopular_6.jpeg")
# Plot the images
fig, axes = plt.subplots(1, 2, figsize=(10, 5))
axes[0].imshow(popular_sample)
axes[0].set_title("Popular image example")
axes[0].axis("off")
axes[1].imshow(unpopular_sample)
axes[1].set_title("Unpopular image example")
axes[1].axis("off")
plt.show()
Next, we split the data into training and test sets.
Training and test sets¶
We use the first eight images in each group as our training data. Hence, our training set consists of 16 images (eight popular, eight unpopular) and our test set contains four images (two popular and two unpopular).
# Training set of 16 images
trainX = [None] * 16
# Select the first 8 images of each group for training
for x in range(8):
trainX[x] = mypic[x] # popular
trainX[x + 8] = mypic[x + 10] # unpopular
# Test set of 4 images
testX = [None] * 4
# Select the last two images in each group for testing, avoiding overlap
for x in range(2):
testX[x] = mypic[x + 8] # popular
testX[x + 2] = mypic[x + 18] # unpopular
Now we go through a few steps of image inspections. We run the below code chunk to understand the image sizes.
# Inspect the structure for resizing
trainX_sizes = [img.size for img in trainX]
testX_sizes = [img.size for img in testX]
print("Train Set Image Sizes:", trainX_sizes)
print("Test Set Image Sizes:", testX_sizes)
Train Set Image Sizes: [(737, 517), (800, 541), (516, 637), (801, 525), (500, 500), (736, 552), (637, 436), (800, 552), (800, 625), (778, 800), (1024, 677), (800, 644), (1025, 599), (799, 574), (799, 533), (800, 533)] Test Set Image Sizes: [(800, 709), (555, 800), (1026, 683), (533, 800)]
The output above suggests that the images are three dimensional but have varying lengths and widths. Looking at the data category of the structure of the list of images, we can see that the first two numbers (width and height) are different for each image and all images are color (‘3’ stands for numbers in RGB format that indicate a color image).
To further prepare the data for training and testing, we want to keep all image dimensions consistent. To do this, we resize all images so that they are 400 by 400 pixels using resize() function. Here, we pick 400 by 400 because it is smaller than the smallest height of all the images in the imported data. Images must also have equal width and height (square dimensions) because we want to end up with square matrices. The below code resizes the images.
# Importing the necessary libraries
from PIL import Image
# Resizing images (train)
trainX_resized = [img.resize((400, 400)) for img in trainX]
# Resizing images (test)
testX_resized = [img.resize((400, 400)) for img in testX]
# Display new dimensions
print("Train Set Image Sizes (After Resizing):", [img.size for img in trainX_resized])
print("Test Set Image Sizes (After Resizing):", [img.size for img in testX_resized])
Train Set Image Sizes (After Resizing): [(400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400), (400, 400)] Test Set Image Sizes (After Resizing): [(400, 400), (400, 400), (400, 400), (400, 400)]
The output shows the size of the images of the resized testing set. We can see that the images now have the same dimensions.
Now we look at the resized images in the training set and test set:
import matplotlib.pyplot as plt
# Display resized images (train)
num_images = len(trainX_resized)
num_cols = 4
num_rows = -(-num_images // num_cols) # Ceiling division to ensure at least num_rows
plt.figure(figsize=(12, 8))
for i, img in enumerate(trainX_resized, 1):
plt.subplot(num_rows, num_cols, i)
plt.imshow(img)
plt.title(f"Train Image {i}")
plt.axis("off")
plt.show()
import matplotlib.pyplot as plt
# Display resized images (test)
num_test_images = len(testX_resized)
num_test_cols = 4
num_test_rows = -(-num_test_images // num_test_cols) # Ceiling division to ensure at least num_test_rows
plt.figure(figsize=(12, 8))
for i, img in enumerate(testX_resized, 1):
plt.subplot(num_test_rows, num_test_cols, i)
plt.imshow(img)
plt.title(f"Test Image {i}")
plt.axis("off")
plt.show()
Next, we install the tensorflow library.
%pip install tensorflow
Requirement already satisfied: tensorflow in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (2.16.1) Requirement already satisfied: tensorflow-intel==2.16.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow) (2.16.1) Requirement already satisfied: absl-py>=1.0.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (2.1.0) Requirement already satisfied: astunparse>=1.6.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (1.6.3) Requirement already satisfied: flatbuffers>=23.5.26 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (24.3.25) Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (0.5.4) Requirement already satisfied: google-pasta>=0.1.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (0.2.0) Requirement already satisfied: h5py>=3.10.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (3.11.0) Requirement already satisfied: libclang>=13.0.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (18.1.1) Requirement already satisfied: ml-dtypes~=0.3.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (0.3.2) Requirement already satisfied: opt-einsum>=2.3.2 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (3.3.0) Requirement already satisfied: packaging in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (23.1) Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (3.20.3) Requirement already satisfied: requests<3,>=2.21.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (2.31.0) Requirement already satisfied: setuptools in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (68.2.2) Requirement already satisfied: six>=1.12.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (1.16.0) Requirement already satisfied: termcolor>=1.1.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (2.4.0) Requirement already satisfied: typing-extensions>=3.6.6 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (4.9.0) Requirement already satisfied: wrapt>=1.11.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (1.14.1) Requirement already satisfied: grpcio<2.0,>=1.24.3 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (1.64.0) Requirement already satisfied: tensorboard<2.17,>=2.16 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (2.16.2) Requirement already satisfied: keras>=3.0.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (3.3.3) Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (0.31.0) Requirement already satisfied: numpy<2.0.0,>=1.23.5 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorflow-intel==2.16.1->tensorflow) (1.26.4) Requirement already satisfied: wheel<1.0,>=0.23.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from astunparse>=1.6.0->tensorflow-intel==2.16.1->tensorflow) (0.41.2) Requirement already satisfied: rich in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from keras>=3.0.0->tensorflow-intel==2.16.1->tensorflow) (13.3.5) Requirement already satisfied: namex in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from keras>=3.0.0->tensorflow-intel==2.16.1->tensorflow) (0.0.8) Requirement already satisfied: optree in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from keras>=3.0.0->tensorflow-intel==2.16.1->tensorflow) (0.11.0) Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.16.1->tensorflow) (2.0.4) Requirement already satisfied: idna<4,>=2.5 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.16.1->tensorflow) (3.4) Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.16.1->tensorflow) (2.0.7) Requirement already satisfied: certifi>=2017.4.17 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.16.1->tensorflow) (2024.2.2) Requirement already satisfied: markdown>=2.6.8 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorboard<2.17,>=2.16->tensorflow-intel==2.16.1->tensorflow) (3.4.1) Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorboard<2.17,>=2.16->tensorflow-intel==2.16.1->tensorflow) (0.7.2) Requirement already satisfied: werkzeug>=1.0.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from tensorboard<2.17,>=2.16->tensorflow-intel==2.16.1->tensorflow) (2.2.3) Requirement already satisfied: MarkupSafe>=2.1.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from werkzeug>=1.0.1->tensorboard<2.17,>=2.16->tensorflow-intel==2.16.1->tensorflow) (2.1.3) Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from rich->keras>=3.0.0->tensorflow-intel==2.16.1->tensorflow) (2.2.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from rich->keras>=3.0.0->tensorflow-intel==2.16.1->tensorflow) (2.15.1) Requirement already satisfied: mdurl~=0.1 in c:\users\gyildiri\appdata\local\anaconda3\lib\site-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->keras>=3.0.0->tensorflow-intel==2.16.1->tensorflow) (0.1.0) Note: you may need to restart the kernel to use updated packages.
Denoting popular images as 1, unpopular images as 0, we label the images in the training and test set as follows:
from tensorflow.keras.utils import to_categorical
# Response Variable for the two categories
trainY = [1] * 8 + [0] * 8
testY = [1, 1, 0, 0]
# One-hot encode labels
trainLabels = to_categorical(trainY)
testLabels = to_categorical(testY)
# Display matrix
print(testLabels)
[[0. 1.] [0. 1.] [1. 0.] [1. 0.]]
The output above shows the matrix of test labels. The first two images are in the popular category, and the next two are in the unpopular category. The same encoding was also done for training labels.
Convolutional neural networks¶
After exploring the general features of the images in our data set, we are ready to implement the CNN modelling approach. The below code chunk is written for setting up the CNN model.
import os
import tensorflow as tf
#Set the TensorFlow random seed.
tf.random.set_seed(2311)
# 4. Configure TensorFlow settings (optional, depending on your TensorFlow version and hardware).
# For TensorFlow 2.x, these settings might help.
os.environ['TF_DETERMINISTIC_OPS'] = '1'
os.environ['TF_CUDNN_DETERMINISTIC'] = '1'
Model configuration¶
Running the code chunk below gives us the model configuration output.
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, Flatten, Dense
import tensorflow as tf
# Model with a linear stack of layers
model = Sequential()
# Layers within the model
model.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', input_shape=(400, 400, 3)))
model.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(units=256, activation='relu'))
model.add(Dropout(rate=0.25))
model.add(Dense(units=2, activation='softmax'))
# Set the optimizer with the adjusted learning rate
optimizer = tf.keras.optimizers.SGD(learning_rate=0.0001, momentum=0.9, decay=1e-6, nesterov=True)
# Compile the model
model.compile(
loss="categorical_crossentropy",
optimizer=optimizer,
metrics=['accuracy']
)
# Display the model summary
model.summary()
Model: "sequential_9"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ conv2d_36 (Conv2D) │ (None, 398, 398, 32) │ 896 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_37 (Conv2D) │ (None, 396, 396, 32) │ 9,248 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_18 (MaxPooling2D) │ (None, 198, 198, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_27 (Dropout) │ (None, 198, 198, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_38 (Conv2D) │ (None, 196, 196, 64) │ 18,496 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_39 (Conv2D) │ (None, 194, 194, 64) │ 36,928 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ max_pooling2d_19 (MaxPooling2D) │ (None, 97, 97, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_28 (Dropout) │ (None, 97, 97, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten_9 (Flatten) │ (None, 602176) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_18 (Dense) │ (None, 256) │ 154,157,312 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_29 (Dropout) │ (None, 256) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_19 (Dense) │ (None, 2) │ 514 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 154,223,394 (588.32 MB)
Trainable params: 154,223,394 (588.32 MB)
Non-trainable params: 0 (0.00 B)
From the output, we can see each layer in our model and the number of parameters that each layer introduces. We explore the model architecture in detail below.
Layers¶
In this model, layers are abbreviated. There are five different layers used:
Convolutional layers (4): the first two convolutional layers use 32 filters with 3 x 3 dimensions. The last two convolutional layers use 64 filters. The primary function of these layers is to extract features from each image. The first two layers extract initial features, and the last two layers include more filters.
Pooling layers (2): pooling layers follow convolutional layers in the VGG architecture. The function of these layers is to reduce the number of parameters in the network by operating on each feature map with a 2 x 2 filter. This helps save space and abstract features to avoid overfitting and is located after two convolutional layers for effective parameter reduction. The type of pooling used is maximum pooling.
Dropout layers (3): the function of these layers is also to decrease the number of parameters to avoid overfitting. In this tutorial, a dropout rate of 0.25 is used, which is 25% of the input units (that are considered weakest predictors) from the previous layer. The input units are set to zero after a pooling layer, thereby reducing parameters even further and freeing up some memory space.
Flattening layer (1): this layer ‘flattens’ the 2 x 2 matrix from the preceding layer into a vector that can be fed into the next layer (a fully connected neural network classifier).
Dense layers (2): these are the fully connected neural network classifiers. The first has 256 neurons, and the last has three neurons.
Activation functions¶
A rectified linear unit (ReLU) is used as an activation function in convolutional layers and in the first fully connected layer. A Softmax activation function is used for the final fully connected layer to produce a probability distribution in output.
Hyperparameters¶
Using the compile() function, we configure the CNN and specify the following parameters:
Loss Function: categorical_crossentropy is used because each image can only belong to one category.
Optimizer: optimizer_sgd() is a stochastic gradient descent optimizer which is currently the best choice for computer vision problems. There are four hyperparameters that this function takes which can be changed to optimize the model:
Before we can train and test our model, we need to convert the images back into the dimensions needed for input into the CNN. The following code chunk reorders the dimensions of the image data in the training and test sets. For example, for the training set, the dimensions are specified as 16 x 400 x 400 x 3, while for the test set, they are 4 x 400 x 400 x 3. The first dimension in the array indicates the number of images, the second and third dimensions refer to the width and height, and the final dimension denotes the colour.
import numpy as np
# Convert each element of the list to an array (if needed)
trainX_arrays = [np.array(img)/ 255.0 for img in trainX_resized]
testX_arrays = [np.array(img)/ 255.0 for img in testX_resized]
# Combine the arrays into a single array with dimensions 16x400x400x3
trainX = np.array(trainX_arrays)
testX = np.array(testX_arrays)
# Print the shape of the arrays to verify the dimensions
print("TrainX Shape:", trainX.shape)
print("TestX Shape:", testX.shape)
TrainX Shape: (16, 400, 400, 3) TestX Shape: (4, 400, 400, 3)
Model training¶
Now that we have built the architecture of the model and the image data is processed, we are ready to integrate the two and train our model. To do this, we use the fit() function from Keras with the training data and training labels. Some defined hyperparameters are:
epochs = 30. This indicates the number times the data gets passed through the CNN for optimisation (this can be changed to optimise the model),
batch_size = 16. This indicates the subset of the data set that will be passed through the CNN at one time and
validation_split = 0.2. This indicates the percentage of training data used for validation source.
# Defined hyperparameters
epochs = 30
batch_size = 16
validation_split = 0.2
# Training the model
history = model.fit(trainX, trainLabels, epochs=epochs, batch_size=batch_size, validation_split=validation_split)
Epoch 1/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 904ms/step - accuracy: 1.0000 - loss: 0.0146 - val_accuracy: 0.0000e+00 - val_loss: 1.9824 Epoch 2/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 853ms/step - accuracy: 1.0000 - loss: 0.0118 - val_accuracy: 0.0000e+00 - val_loss: 1.9915 Epoch 3/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 851ms/step - accuracy: 1.0000 - loss: 0.0143 - val_accuracy: 0.0000e+00 - val_loss: 1.9962 Epoch 4/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 845ms/step - accuracy: 1.0000 - loss: 0.0168 - val_accuracy: 0.0000e+00 - val_loss: 2.0035 Epoch 5/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 844ms/step - accuracy: 1.0000 - loss: 0.0161 - val_accuracy: 0.0000e+00 - val_loss: 1.9978 Epoch 6/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 889ms/step - accuracy: 1.0000 - loss: 0.0191 - val_accuracy: 0.0000e+00 - val_loss: 1.9906 Epoch 7/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 892ms/step - accuracy: 1.0000 - loss: 0.0144 - val_accuracy: 0.2500 - val_loss: 1.9867 Epoch 8/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 887ms/step - accuracy: 1.0000 - loss: 0.0114 - val_accuracy: 0.2500 - val_loss: 1.9788 Epoch 9/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 840ms/step - accuracy: 1.0000 - loss: 0.0108 - val_accuracy: 0.2500 - val_loss: 1.9819 Epoch 10/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 814ms/step - accuracy: 1.0000 - loss: 0.0122 - val_accuracy: 0.2500 - val_loss: 1.9850 Epoch 11/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 861ms/step - accuracy: 1.0000 - loss: 0.0258 - val_accuracy: 0.2500 - val_loss: 1.9811 Epoch 12/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 823ms/step - accuracy: 1.0000 - loss: 0.0159 - val_accuracy: 0.2500 - val_loss: 1.9681 Epoch 13/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 789ms/step - accuracy: 1.0000 - loss: 0.0124 - val_accuracy: 0.2500 - val_loss: 1.9746 Epoch 14/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 827ms/step - accuracy: 1.0000 - loss: 0.0105 - val_accuracy: 0.2500 - val_loss: 1.9753 Epoch 15/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 820ms/step - accuracy: 1.0000 - loss: 0.0144 - val_accuracy: 0.2500 - val_loss: 1.9711 Epoch 16/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 842ms/step - accuracy: 1.0000 - loss: 0.0114 - val_accuracy: 0.2500 - val_loss: 1.9854 Epoch 17/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 841ms/step - accuracy: 1.0000 - loss: 0.0107 - val_accuracy: 0.2500 - val_loss: 1.9968 Epoch 18/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 830ms/step - accuracy: 1.0000 - loss: 0.0095 - val_accuracy: 0.2500 - val_loss: 2.0103 Epoch 19/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 834ms/step - accuracy: 1.0000 - loss: 0.0124 - val_accuracy: 0.2500 - val_loss: 2.0129 Epoch 20/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 827ms/step - accuracy: 1.0000 - loss: 0.0082 - val_accuracy: 0.2500 - val_loss: 2.0253 Epoch 21/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 831ms/step - accuracy: 1.0000 - loss: 0.0134 - val_accuracy: 0.2500 - val_loss: 2.0249 Epoch 22/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 855ms/step - accuracy: 1.0000 - loss: 0.0089 - val_accuracy: 0.0000e+00 - val_loss: 2.0370 Epoch 23/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 846ms/step - accuracy: 1.0000 - loss: 0.0089 - val_accuracy: 0.0000e+00 - val_loss: 2.0446 Epoch 24/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 876ms/step - accuracy: 1.0000 - loss: 0.0099 - val_accuracy: 0.0000e+00 - val_loss: 2.0571 Epoch 25/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 876ms/step - accuracy: 1.0000 - loss: 0.0109 - val_accuracy: 0.0000e+00 - val_loss: 2.0611 Epoch 26/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 900ms/step - accuracy: 1.0000 - loss: 0.0086 - val_accuracy: 0.0000e+00 - val_loss: 2.0679 Epoch 27/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 846ms/step - accuracy: 1.0000 - loss: 0.0148 - val_accuracy: 0.0000e+00 - val_loss: 2.0621 Epoch 28/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 877ms/step - accuracy: 1.0000 - loss: 0.0105 - val_accuracy: 0.0000e+00 - val_loss: 2.0532 Epoch 29/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 858ms/step - accuracy: 1.0000 - loss: 0.0094 - val_accuracy: 0.0000e+00 - val_loss: 2.0464 Epoch 30/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 897ms/step - accuracy: 1.0000 - loss: 0.0093 - val_accuracy: 0.2500 - val_loss: 2.0429
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
epochs = range(1, len(history.history['accuracy']) + 1)
# Scatter plot and trend line for loss
plt.figure(figsize=(7, 4))
sns.scatterplot(x=epochs, y=history.history['loss'], label='Training Loss', color='Coral')
sns.scatterplot(x=epochs, y=history.history['val_loss'], label='Validation Loss', color='Turquoise')
sns.lineplot(x=epochs, y=history.history['loss'], color='Coral')
sns.lineplot(x=epochs, y=history.history['val_loss'], color='Turquoise')
plt.title('Training and Validation Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()
# Scatter plot and trend line for accuracy
plt.figure(figsize=(7, 4))
sns.scatterplot(x=epochs, y=history.history['accuracy'], label='Training Accuracy', color='Coral')
sns.scatterplot(x=epochs, y=history.history['val_accuracy'], label='Validation Accuracy', color='Turquoise')
sns.lineplot(x=epochs, y=history.history['accuracy'], color='Coral')
sns.lineplot(x=epochs, y=history.history['val_accuracy'], color='Turquoise')
plt.title('Training and Validation Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
plt.show()
As the model trains epoch after epoch, cross-entropy (‘loss’) should be minimized in both training and validation splits. If there is an apparent increase in the validation split after a certain epoch, then the plot is suggestive of slight overfitting.
The figure above shows the model fit output with the loss and accuracy metrics. The training loss is minimal and stable over time, while the validation loss decreases and then increases slightly, suggesting potential overfitting. The training accuracy remains high at around 90%, but the validation accuracy stays close to 0%. This indicates that the model performs well on the training data but fails to generalize to the validation data. This is due to the fact that we used only a few images to train the model, i.e. the model is not able to learn much.
Model performance evaluation¶
We use the evaluate() function to calculate loss and accuracy metrics of the above model. The below code chunk evaluates the model performance and produces the corresponding output.
# Evaluate the model on the training set
eval_train = model.evaluate(trainX, trainLabels)
eval_train
# Print the evaluation metrics
print("Loss and Accuracy on Training Set:")
print("Loss:", eval_train[0])
print("Accuracy:", eval_train[1])
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 198ms/step - accuracy: 0.8125 - loss: 0.5272 Loss and Accuracy on Training Set: Loss: 0.5271610021591187 Accuracy: 0.8125
This model has had a 81.25 percent accuracy for predicting the training set with a loss of 52.72 percent.
Further models¶
The model we trained above needs to be run more than once to get the highest accuracy. Below we re-run the code chunk and evaluate the accuracy values. Please note that the results may vary each time these models are run.
Run 2:¶
# Defined hyperparameters
epochs = 30
batch_size = 16
validation_split = 0.2
# Training the model
history2 = model.fit(trainX, trainLabels, epochs=epochs, batch_size=batch_size, validation_split=validation_split)
Epoch 1/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2814 - val_accuracy: 0.0000e+00 - val_loss: 1.0866 Epoch 2/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.3021 - val_accuracy: 0.0000e+00 - val_loss: 1.0933 Epoch 3/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 0.9167 - loss: 0.2845 - val_accuracy: 0.0000e+00 - val_loss: 1.0935 Epoch 4/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2656 - val_accuracy: 0.0000e+00 - val_loss: 1.0991 Epoch 5/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2933 - val_accuracy: 0.0000e+00 - val_loss: 1.0987 Epoch 6/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2647 - val_accuracy: 0.0000e+00 - val_loss: 1.1079 Epoch 7/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2665 - val_accuracy: 0.0000e+00 - val_loss: 1.1113 Epoch 8/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 0.9167 - loss: 0.2813 - val_accuracy: 0.0000e+00 - val_loss: 1.1101 Epoch 9/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2696 - val_accuracy: 0.2500 - val_loss: 1.1075 Epoch 10/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2444 - val_accuracy: 0.2500 - val_loss: 1.1035 Epoch 11/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2365 - val_accuracy: 0.2500 - val_loss: 1.1144 Epoch 12/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2111 - val_accuracy: 0.2500 - val_loss: 1.1302 Epoch 13/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2215 - val_accuracy: 0.2500 - val_loss: 1.1498 Epoch 14/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2069 - val_accuracy: 0.0000e+00 - val_loss: 1.1747 Epoch 15/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2181 - val_accuracy: 0.0000e+00 - val_loss: 1.1971 Epoch 16/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2178 - val_accuracy: 0.0000e+00 - val_loss: 1.2160 Epoch 17/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2031 - val_accuracy: 0.0000e+00 - val_loss: 1.2349 Epoch 18/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2104 - val_accuracy: 0.0000e+00 - val_loss: 1.2391 Epoch 19/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.1887 - val_accuracy: 0.0000e+00 - val_loss: 1.2475 Epoch 20/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2116 - val_accuracy: 0.0000e+00 - val_loss: 1.2535 Epoch 21/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2097 - val_accuracy: 0.2500 - val_loss: 1.2369 Epoch 22/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2122 - val_accuracy: 0.2500 - val_loss: 1.2111 Epoch 23/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.2008 - val_accuracy: 0.2500 - val_loss: 1.1907 Epoch 24/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.1807 - val_accuracy: 0.2500 - val_loss: 1.1824 Epoch 25/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.1840 - val_accuracy: 0.2500 - val_loss: 1.1938 Epoch 26/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.1783 - val_accuracy: 0.2500 - val_loss: 1.1845 Epoch 27/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.1704 - val_accuracy: 0.2500 - val_loss: 1.1984 Epoch 28/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.1576 - val_accuracy: 0.2500 - val_loss: 1.2161 Epoch 29/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.1499 - val_accuracy: 0.2500 - val_loss: 1.2533 Epoch 30/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.1428 - val_accuracy: 0.2500 - val_loss: 1.2687
# Scatter plot and trend line for loss
epochs = range(1, len(history2.history['accuracy']) + 1)
plt.figure(figsize=(7, 4))
sns.scatterplot(x=epochs, y=history2.history['loss'], label='Training Loss', color='Coral')
sns.scatterplot(x=epochs, y=history2.history['val_loss'], label='Validation Loss', color='Turquoise')
sns.lineplot(x=epochs, y=history2.history['loss'], color='Coral')
sns.lineplot(x=epochs, y=history2.history['val_loss'], color='Turquoise')
plt.title('Training and Validation Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()
# Scatter plot and trend line for accuracy
plt.figure(figsize=(7, 4))
sns.scatterplot(x=epochs, y=history2.history['accuracy'], label='Training Accuracy', color='Coral')
sns.scatterplot(x=epochs, y=history2.history['val_accuracy'], label='Validation Accuracy', color='Turquoise')
sns.lineplot(x=epochs, y=history2.history['accuracy'], color='Coral')
sns.lineplot(x=epochs, y=history2.history['val_accuracy'], color='Turquoise')
plt.title('Training and Validation Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
plt.show()
# Evaluate the model on the training set
eval_train = model.evaluate(trainX, trainLabels)
eval_train
# Print the evaluation metrics
print("Loss and Accuracy on Training Set:")
print("Loss:", eval_train[0])
print("Accuracy:", eval_train[1])
1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 822ms/step - accuracy: 0.8125 - loss: 0.4571 Loss and Accuracy on Training Set: Loss: 0.45706281065940857 Accuracy: 0.8125
Run 3:¶
Please note that the results may vary.
# Defined hyperparameters
epochs = 30
batch_size = 16
validation_split = 0.2
# Training the model
history3 = model.fit(trainX, trainLabels, epochs=epochs, batch_size=batch_size, validation_split=validation_split)
Epoch 1/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0652 - val_accuracy: 0.2500 - val_loss: 1.5312 Epoch 2/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0722 - val_accuracy: 0.2500 - val_loss: 1.5258 Epoch 3/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0746 - val_accuracy: 0.2500 - val_loss: 1.5244 Epoch 4/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0521 - val_accuracy: 0.2500 - val_loss: 1.5345 Epoch 5/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0640 - val_accuracy: 0.2500 - val_loss: 1.5338 Epoch 6/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0440 - val_accuracy: 0.2500 - val_loss: 1.5459 Epoch 7/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0598 - val_accuracy: 0.2500 - val_loss: 1.5627 Epoch 8/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0414 - val_accuracy: 0.2500 - val_loss: 1.5768 Epoch 9/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0646 - val_accuracy: 0.2500 - val_loss: 1.5969 Epoch 10/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0585 - val_accuracy: 0.2500 - val_loss: 1.5965 Epoch 11/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0651 - val_accuracy: 0.2500 - val_loss: 1.5912 Epoch 12/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0550 - val_accuracy: 0.2500 - val_loss: 1.5894 Epoch 13/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0426 - val_accuracy: 0.2500 - val_loss: 1.5920 Epoch 14/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0387 - val_accuracy: 0.2500 - val_loss: 1.6084 Epoch 15/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0456 - val_accuracy: 0.2500 - val_loss: 1.6189 Epoch 16/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0590 - val_accuracy: 0.2500 - val_loss: 1.6105 Epoch 17/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0445 - val_accuracy: 0.2500 - val_loss: 1.6258 Epoch 18/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0443 - val_accuracy: 0.2500 - val_loss: 1.6464 Epoch 19/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0418 - val_accuracy: 0.2500 - val_loss: 1.6614 Epoch 20/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0415 - val_accuracy: 0.2500 - val_loss: 1.6696 Epoch 21/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0372 - val_accuracy: 0.2500 - val_loss: 1.6815 Epoch 22/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0425 - val_accuracy: 0.2500 - val_loss: 1.6869 Epoch 23/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0337 - val_accuracy: 0.2500 - val_loss: 1.7018 Epoch 24/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0534 - val_accuracy: 0.2500 - val_loss: 1.7368 Epoch 25/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0317 - val_accuracy: 0.0000e+00 - val_loss: 1.7555 Epoch 26/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0436 - val_accuracy: 0.0000e+00 - val_loss: 1.7695 Epoch 27/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0321 - val_accuracy: 0.0000e+00 - val_loss: 1.7814 Epoch 28/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0323 - val_accuracy: 0.0000e+00 - val_loss: 1.7864 Epoch 29/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0635 - val_accuracy: 0.2500 - val_loss: 1.7332 Epoch 30/30 1/1 ━━━━━━━━━━━━━━━━━━━━ 3s 3s/step - accuracy: 1.0000 - loss: 0.0421 - val_accuracy: 0.2500 - val_loss: 1.6930
epochs = range(1, len(history3.history['accuracy']) + 1)
# Scatter plot and trend line for loss
plt.figure(figsize=(7, 4))
sns.scatterplot(x=epochs, y=history3.history['loss'], label='Training Loss', color='Coral')
sns.scatterplot(x=epochs, y=history3.history['val_loss'], label='Validation Loss', color='Turquoise')
sns.lineplot(x=epochs, y=history3.history['loss'], color='Coral')
sns.lineplot(x=epochs, y=history3.history['val_loss'], color='Turquoise')
plt.title('Training and Validation Loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()
# Scatter plot and trend line for accuracy
plt.figure(figsize=(7, 4))
sns.scatterplot(x=epochs, y=history3.history['accuracy'], label='Training Accuracy', color='Coral')
sns.scatterplot(x=epochs, y=history3.history['val_accuracy'], label='Validation Accuracy', color='Turquoise')
sns.lineplot(x=epochs, y=history3.history['accuracy'], color='Coral')
sns.lineplot(x=epochs, y=history3.history['val_accuracy'], color='Turquoise')
plt.title('Training and Validation Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
plt.show()
# Evaluate the model on the training set
eval_train = model.evaluate(trainX, trainLabels)
eval_train
# Print the evaluation metrics
print("Loss and Accuracy on Training Set:")
print("Loss:", eval_train[0])
print("Accuracy:", eval_train[1])
1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 823ms/step - accuracy: 0.8125 - loss: 0.4675 Loss and Accuracy on Training Set: Loss: 0.4675377607345581 Accuracy: 0.8125
The model always initializes with different parameter values. On our second and third try, for instance, this model has had a 81.25 percent accuracy for predicting the training set with a loss of 46.75 percent.
Before going to the test set, we can use the model to make a prediction of the training set and create a confusion matrix. We can then evaluate the model by looking at the probability that the model assigned a certain category to each image. The code chunk below is for this task.
import numpy as np
import pandas as pd
# Make predictions on the training data
predTrain = model.predict(trainX)
# Create an array to store predicted classes
pred_class = np.zeros(16)
# Populate the predicted class array based on the predicted probabilities
for i in range(16):
if predTrain[i, 1] > 0.5:
pred_class[i] = 1
else:
pred_class[i] = 0
# Round the predicted probabilities to two decimal places
probability_popular = np.round(predTrain[:, 1], 2)
# Create a DataFrame to store the results
results_summary = pd.DataFrame({
"predict_probability_1": probability_popular,
"predicted_class": pred_class,
"actual_class": trainY
})
# Print the results summary
print(results_summary)
# Visualize the confusion matrix
confusion_matrix = pd.crosstab(pred_class, trainY, rownames=['Predicted'], colnames=['Actual'])
print(confusion_matrix)
1/1 ━━━━━━━━━━━━━━━━━━━━ 1s 908ms/step predict_probability_1 predicted_class actual_class 0 0.98 1.0 1 1 0.98 1.0 1 2 0.81 1.0 1 3 0.98 1.0 1 4 0.99 1.0 1 5 0.99 1.0 1 6 0.97 1.0 1 7 0.98 1.0 1 8 0.05 0.0 0 9 0.05 0.0 0 10 0.15 0.0 0 11 0.10 0.0 0 12 0.77 1.0 0 13 0.81 1.0 0 14 0.95 1.0 0 15 0.48 0.0 0 Actual 0 1 Predicted 0.0 5 0 1.0 3 8
In the 'prediction of classes' output, the first column refers to the probability that an image is classified as class '1', i.e., popular. The next two columns show the predicted and the actual class of each image respectively.
The confusion matrix output summarizes the model predictions. Class '0' refers to the unpopular images while class '1' refers to the popular images. The column shows the actual class (0 or 1) and the rows show the predicted class (0 or 1). Note that one could simply switch rows and columns of the confusion matrix, so always watch out for the row and column labels. The diagonal entries in this matrix show the number of correctly classified images. For example, there were eight popular images in the training set, and the model predicted the class of all these images correctly. Three out of eight unpopular images were classified incorrectly (as popular while it was actually unpopular).
Now we turn to the test set and obtain class predictions.
Prediction using the test set¶
Now we repeat the above procedure for the test set.
import numpy as np
import pandas as pd
# Evaluate the model on the test set
evTest = model.evaluate(testX, testLabels)
print(evTest)
# Generate predicted probabilities for the test set
predTest = model.predict(testX)
# Create an array to store predicted classes
pred_class = np.zeros(4)
# Populate the predicted class array based on the predicted probabilities
for i in range(4):
if predTest[i, 1] > 0.5:
pred_class[i] = 1
else:
pred_class[i] = 0
# Round the predicted probabilities to two decimal places
probability_popular_test = np.round(predTest[:, 1], 2)
# Create a DataFrame to store the results
results_summary = pd.DataFrame({
"predict_probability_1": probability_popular_test,
"predicted_class": pred_class,
"actual_class": testY
})
# Print the results summary
print(results_summary)
# Visualize the confusion matrix
confusion_matrix = pd.crosstab(pred_class, testY, rownames=['Predicted'], colnames=['Actual'])
print(confusion_matrix)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 158ms/step - accuracy: 0.5000 - loss: 0.6165 [0.6165364980697632, 0.5] 1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 92ms/step predict_probability_1 predicted_class actual_class 0 1.00 1.0 1 1 1.00 1.0 1 2 0.68 1.0 0 3 0.73 1.0 0 Actual 0 1 Predicted 1.0 2 2
The output suggests that the accuracy of prediction based on the test set is 50 per cent. Specifically, the model failed to predict the class for two of the images in the test set. With larger data sets, we will train the CNN model further to improve model predictions for the testing set.
In this software application, we show that the keras package provides a simple way to implement multiple types of CNN architectures and facilitates easy fine-tuning of hyperparameters. To find a CNN model that produces robust predictions, we need to find a good balance of accuracy and loss for both training and test sets. To build such a model, there are a few things we can do:
Increase the number of images in our training and test set. This would also require a graphic processing unit. Otherwise, it just might take a very long time to run.
Add more diverse images with varying colours, tones, and backgrounds.
Decrease loss by adding more convolutional layers and fully connected layers.
Decrease the learning rate.
What makes an attractive image?¶
Assuming that our CNN is the most accurate that we can possibly have, what managerial conclusions can we draw from the model? Focusing on the images that are classified as ‘popular’ in the test set, we can investigate the features of those images. In doing so, the brand can design its marketing communications more effectively. Note that the testing set here is very limited, consisting of only four images with three classified as ‘popular’. The interpretations will be much more accurate and convincing if you have a larger test set.
The code chunk below examines four features in the images: (i) color, (ii) human-centric, (iii) direct logo exposure, and (iv) direct product exposure.
import matplotlib.pyplot as plt
testset = data.iloc[[8, 9, 18]]
# Convert categorical columns to factors
testset['Colour Image'] = testset['Colour Image'].astype('category')
testset['Human Centric'] = testset['Human Centric'].astype('category')
testset['Direct Logo Exposure'] = testset['Direct Logo Exposure'].astype('category')
testset['Direct Product Exposure'] = testset['Direct Product Exposure'].astype('category')
# Plot histograms
plt.figure(figsize=(8, 6))
plt.subplot(2, 2, 1)
plt.hist(testset['Colour Image'], color='blue', alpha=0.6, bins=20, edgecolor='black')
plt.xlabel('Color Image')
plt.ylabel('Count')
plt.title('Color Image')
plt.subplot(2, 2, 2)
plt.hist(testset['Human Centric'], color='green', alpha=0.6, bins=20, edgecolor='black')
plt.xlabel('Human Centric')
plt.ylabel('Count')
plt.title('Human Centric')
plt.subplot(2, 2, 3)
plt.hist(testset['Direct Logo Exposure'], color='red', alpha=0.6, bins=20, edgecolor='black')
plt.xlabel('Direct Logo Exposure')
plt.ylabel('Count')
plt.title('Direct Logo Exposure')
plt.subplot(2, 2, 4)
plt.hist(testset['Direct Product Exposure'], color='yellow', alpha=0.6, bins=20, edgecolor='black')
plt.xlabel('Direct Product Exposure')
plt.ylabel('Count')
plt.title('Direct Product Exposure')
plt.tight_layout()
plt.show()
The output above shows the number of times these features appear in the popular images from the test set. Judging from the results above, consumers, in general, will be engaged by images that are in colour, with humans, with the brand logo displayed and probably with the product also displayed.
Here, we have followed an intuitive approach to analyse image features. A more comprehensive approach would be to run a conjoint analysis using the extracted image features from the CNN model. However, one should note that model-based image features may not always be interpretable.
Summary¶
Consumers increasingly take, store, and share images on social media and other digital platforms. In parallel, marketers have more analytical tools than ever. Both the abundance of image data and the recent developments in deep learning modelling techniques open new opportunities for marketers to plan and execute their branding, advertising, product design and presentation strategies. For example, by identifying the key features of user-generated images (e.g., colours, objects, logos) shared on social media, marketing managers can monitor ‘visual conversations’, follow trends, and increase the effectiveness of their marketing mix decisions. In this chapter, we focus on image analytics through convolutional neural networks (CNNs), a cutting-edge deep learning model that has recently gained popularity among marketing practitioners and academics. CNNs are used to process and classify images. We apply the CNN modelling tool to a sample of consumer-uploaded images of Coca-Cola on Flickr to illustrate how these models can be applied to real-world data sets.
Although CNNs are very useful for image classification problems, their applications sometimes are challenging. For example, important objects in user-generated images may appear partially or distorted, which complicates the model learning. Also, images that contain brand images may be shown as a combination of text and visuals rather than just visuals. In such cases, extracting image features becomes very complex. Moreover, brand images may appear in different contexts. A user may share an image with a famous football player who wears a Nike t-shirt. Understanding the consumer motives behind such an image sharing behaviour is difficult if we rely only on image classification algorithms. Therefore, practitioners are recommended to complement deep learning techniques with behavioural research methods.